Method for generating simulations of fluid interfaces for improved animation of fluid interactions

ABSTRACT

A method for generating visual representations of interactions between two different materials is provided. The method can be performed using a computing device operated by a computer user or artist. The method includes modeling a primary material as a plurality of first particles and modeling a layer portion of a secondary material as a fluid volume. The secondary material can include a layer portion positioned between the plurality of first particles and an outer portion. At least one boundary condition might be assigned to a boundary positioned between the layer portion and the outer portion, the at least one boundary condition includes at least one pressure value. Values of motion parameters might be determined by applying the at least one boundary condition at the boundary and generating one or more visual representations of the primary material interacting with the secondary material based on the values of the motion parameters.

CROSS-REFERENCES TO PRIORITY AND RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/183,993, filed Feb. 21, 2020, and claims the benefit of U.S.Provisional Application No. 62/983,435 filed Feb. 28, 2020, which isincorporated by reference in its entirety for all purposes.

FIELD OF THE INVENTION

The present disclosure generally relates to simulating interactionsbetween different materials, and more particularly to efficientcomputational approaches for simulation of interactions betweendifferent materials.

BACKGROUND

Visual representations of scenes intended to reflect real-worldscenarios are common in animation and other fields. For example, acomputer-generated imagery scene could be created by having an artistmanually draw a sequence of frames to form a video sequence. For simplecartoons, for example, this is a feasible approach. However, as viewershave come to expect more complex visuals, there is a need forcomputer-driven imagery generation. Some of that computer-driven imagerygeneration might rely on simulation.

Computer simulation that is used for imagery generation has been used toanimate natural phenomena as well as natural movements of characters,such as by using a physics engine to output movements of an articulatedcharacter that are consistent with real-world physics and jointconstraints. In some ways, this is often a simple problem—how todetermine natural-looking movements of at most a few dozen attached bodyparts. For other simulations, such as those with flexible objects,fluids, and the like, the number of degrees of freedom of individualunits is much greater and typically computer simulation requires atrade-off between realism, resolution, and an amount of computingresources available. Because of this trade-off, efficient computersimulation techniques can be important as they might allow for anincrease in realism and/or resolution without requiring significantincreases in computing resources. Simulation computations involvingbubbles, waterfalls, and other fluid interactions can often involve suchtrade-offs.

Fluid simulation is ubiquitous in computer graphics. When there is onlya single fluid (or gas) of interest, practitioners typically useconventional single-phase fluid simulation tools to determine thefluid's motion. This means the area outside of the fluid is treated as avacuum. But, multiple fluids are often present and cannot be adequatelysimulated using conventional single-phase fluid simulation tools. Forinstance, a waterfall looks significantly different when the water fallsthrough vacuum instead of air. Similarly, an underwater air bubble wouldcollapse if the bubble is represented as a vacuum, which is clearly notcase for a real-world air bubble. In these examples, air needs to beaccounted for and not modeled as being a vacuum, to achieve the properlook of the interaction between the air and water. As such, thistypically involves a two-phase air-water coupled simulation.Unfortunately, such two-phase air-water coupled simulations aretypically quite computationally expensive to perform.

Therefore, there is a need for a more efficient approach to performingsimulations of interactions between different materials, that can beapplicable to, for example, two-phase air-water coupled simulations.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 is a diagram of a data flow through a system when the system isgenerating values of motion parameters, which are used to create visualrepresentations of a first material interacting with a second material.

FIG. 2 is a flowchart of the process of generating the values of themotion parameters.

FIG. 3 illustrates a primary material surrounded by a secondarymaterial.

FIG. 4 illustrates a drag force exchange that may be used to couple theprimary material and the secondary material together after a first setof equations has been solved for the primary material and a second setof equations has been solved separately for the secondary material.

FIG. 5 illustrates a method of calculating a new velocity field as afunction of a previous velocity field, an aeration field, and a dragforce field.

FIG. 6 illustrates an example visual content generation system as mightbe used to generate imagery in the form of still images and/or videosequences of images, according to various embodiments.

FIG. 7 is a block diagram illustrating an example computer system uponwhich computer systems of the systems illustrated in FIGS. 1 and 6 maybe implemented.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

In a computer simulation involving three dimensions and having an outputthat is imagery (such as a still image or a sequence of video frames),often the virtual objects and material being simulated are representedrelative to a three-dimensional (“3D”) grid in a virtual space with thegrid being divided into voxels. Some elements might have sub-voxelresolution.

In typical computer simulations, it is difficult to achieve realisticlooking sceneries that comprise moving objects, e.g., waterfalls andunderwater bubbles. Waterfall simulations typically involved draggingwater towards a prescribed, artistically driven air field. In suchsimulations, the air affects the water, but the water does not affectthe air. Another approach represents the air as a single velocity field,and (partially) applies a divergence-free projection to the singlevelocity field. This approach gives the appearance that the water hasaffected the air and may create an interesting flow of air that in turnaffects the water. However, it is unclear to what degree such solutionsare physics based. Earlier simulation techniques for simulatingunderwater bubbles include, for example, R. Goldade and C. Batty,Constraint bubbles: Adding efficient zero-density bubbles toincompressible free surface flow, 2017 adopt a particle-in-cell fluidsimulator that represents each air pocket as a volume conserving voidwith fixed pressure. While such technique is capable of recreatingrealistic gargling water effects, it does not capture subtle bubbledetail that makes it fully realistic. By way of another example, L. Boydand R. Bridson, Multiflip for energetic two-phase fluid simulation, ACMTrans. Graph., 31(2), April 2012, use a Fluid Implicit Particle (“FLIP”)method to discretize both water and air and perform a two-phaseincompressible solve.

On the other hand, bubbles smaller than a grid voxel size are typicallyrepresented as a separate particle system. For example, D. Kim, O. Song,and H. Ko, A practical simulation of dispersed bubble flow, ACM Trans.Graph., 29(4), July 2010, passively advect those particles with the bulkfluid and use them to adjust effective density of water, leading tonaturalistic buoyancy effects. They employ a stochastic solver foradditional sub-voxel motion. By way of another example, S. Patkar, M.Aanjaneya, D. Karpman, and R. Fedkiw, A hybrid lagrangian-eulerianformulation for bubble generation and dynamics, In Proc. of the ACMSIGGRAPH/Eur. Symp. on Comp. Anim., SCA, pages 105-114, New York, N.Y.,USA, 2013, ACM, use an Eulerian two-phase approach for simulatingbubbles larger than the grid voxel size and passively advected particlesfor tracking bubbles smaller than the grid voxel size. Patkar et al.combine the two differently sized groups of bubbles in a single linearsolve, which also handles compressibility.

A typical method of simulating bubbles (e.g., primary material 302 ofFIG. 3) moving deep under water (e.g., secondary material 304) creates aFluid Implicit Particle (“FLIP”) model of the water and represents thebubbles as constraints. An example of this approach is provided bySideFX Houdini software. This approach concentrates most of thecomputational resources on the water and preserves its volume.Unfortunately, tracking the bubbles and preserving their volume is aproblem because they are not modeled as a full phase. This makes thebubble movement with respect to the water less realistic, which isunfortunate because the bubbles are more visual significant than thewater.

Another method of simulating bubbles uses a FLIP model to simulate boththe bubbles (e.g., the primary material 302) and the water (e.g., thesecondary material 304). An example of this approach is provided by L.Boyd and R. Bridson, Multiflip for energetic two-phase fluid simulation,ACM Trans. Graph., 31(2), April 2012. Using this approach, both thewater and the bubbles are accurately represented. Unfortunately, thisapproach is computationally expensive because it fully represents all ofthe water.

Via various embodiments, more efficient simulation approaches that canprovide the same high level of realistic looking interactions betweendifferent materials are presented. These efficient simulations areperformed without partaking computationally expensive approaches thatincur in traditional approaches that fully take into account all of thewater.

In many of the examples described herein, inputs to a computersimulation system include parameters about the virtualmaterial/object/fluid/etc. being simulated and an output of a computersimulation are the positions/mass/movement/etc. of the virtualmaterial/object/fluid/etc. Such an output might be an input to ananimation system, which can provide for rendering computer-generatedimagery of the virtual material/object/fluid/etc. present in a scene ina virtual space. The computer-generated imagery might be still images,stereoscopic images, video sequences, and/or stereoscopic videosequences. In some cases, the computer simulation of virtual elementsseeks to match what would happen with corresponding real-world elements,but in other cases, artistic or other inputs are used in the computersimulation to create effects that do not correspond to anything in thereal-world, or at least anything in available physical environments. Forexample, in a given simulation, an operator of a simulation engine mightprovide an input that corresponds to gravity “turning off” for a shortperiod of time, which can be simulated but has no real-worldcorrespondence.

The primary material may be modeled as a plurality of particles orobjects that may, in some cases, be unconstrained relative to oneanother, such that each object can move independently of the others.This may occur for example with granular media such as droplets orbubbles, and may be thought of as a zero-dimensional constraint, or aconstraint on zero degrees of freedom. A one-dimensional constraint, orconstraint of a single degree of freedom, may occur for example withhair, wherein the hairs are free to move relative to one another alongmost of their lengths, but are fixed at one end relative to one another.A two-dimensional constraint or two-degree-of-freedom constraint may forexample occur with cloth, wherein the objects of the porous medium areinterwoven fibers that are free to move, bend, or fold in threedimensions but have fixed locations relative to one another within thetopological plane of the cloth. A three-dimensional constraint orthree-degree-of-freedom constraint can occur for example with athree-dimensional network such as a sponge, wherein the objects of theporous medium are fibers or other shapes that intertwine in threedimensions. A sponge may be capable of bending or flexing, but theobjects making up the sponge may have fixed spatial relationships to oneanother within the topological volume of the sponge. In some cases,coupling or constraint between two fluid objects, or objects within afluid, may occur through surface tension.

FIG. 1 is a diagram of a data flow through a system 100 when the system100 is configured to perform a process 200 (see FIG. 2) that generatesvalues of motion parameters 110. The motion parameters 110 are used byan animation creation system 630, which is a component of an examplevisual content generation system 600 (see FIG. 6), to create visualrepresentations of interactions between first material 112 and secondmaterial 114. For example, the system 100 may be used to simulate one ormore bubbles of the first material 112 positioned inside (e.g., floatingwithin) the second material 114. The first material 112 and the secondmaterial 114 are different materials and each may represent a gas, acombination of gases (e.g., air), a liquid, another fluid, or acombination of fluids. Additionally, the first material 112 and thesecond material 114 may include solid particles held in suspension orfloating therein.

In some embodiments, the first material 112 and the second material 114may be configured to remain separate, at least temporarily, when mixedtogether. By way of a non-limiting example, the first material 112 maybe air and the second material 114 may be water, or vice versa. When oneof the first material 112 and the second material 114 is a gas and theother is a liquid, the system 100 may be characterized as simulatinginteractions between multiple phases of matter, namely gas and liquidphases. The system 100 may also be used to simulate the first material112 and the second material 114 in the same phase. For example, one ofthe first material 112 and the second material 114 may be a polar fluid(e.g., water) and the other may be a non-polar fluid (e.g., oil).

Referring to FIG. 1, the system 100 as shown includes a motionsimulation system 120 and at least one client computing device 140operated by at least one human artist 142. The motion simulation system120 may be implemented by software executing on one or more computersystems (e.g., each like a computer system 700 illustrated in FIG. 7).The motion simulation system 120 is configured to receive data definingthe first material 112 and data defining the second material 114, whichare used to output the values of the motion parameters 110. The motionsimulation system 120 may be implemented as a fluid simulator (e.g., aparticle-in-cell fluid simulator) configured to strongly couple thefirst material 112 and the second material 114 together by solving a setof equations for the first material 112 and the second material 114 atthe same time. For example, the motion simulation system 120 may beconfigured to perform a two-phase pressure solve on the first material112 and the second material 114 at the same time. The values of themotion parameters 110 may include the solution obtained for the set ofequations. The two-phase pressure solve may be an incompressibletwo-phase Navier-Stokes solve on an Eulerian grid (also referred to as atwo-phase incompressible ghost-fluid Eulerian solve). Alternatively,and/or additionally, the motion simulation system 120 may be configuredto weakly or iteratively couple the first material 112 and the secondmaterial 114 together by separately solving a first set of equations forthe first material 112 and a second set of equations for the secondmaterial 114. The values of the motion parameters 110 may include thesolutions obtained for the first and second sets of equations. After oneof the first and second sets of equations is solved, the solution may besupplied to the other set of equations. Further, as explained below, themotion simulation system 120 may be configured to weakly or iterativelycouple the solutions together (e.g., with a drag force 404 illustratedin FIG. 4).

The values of the motion parameters 110 may include the data definingthe first material 112 and the data defining the second material 114.The values of the motion parameters 110 may be generated based at leastin part on parameter values 144 that may include parameter valuesdefined by the artist 142 (e.g., using the client computing device 140)and/or parameter values that are predetermined and stored in a datastore. When the parameter values 144 include user-defined parametervalues, the motion of the first material 112 and/or the second material114 may be characterized as being at least partially art directable.

As described below, the visual content generation system 600 (see FIG.6) is configured to receive the values of the motion parameters 110 asinput, and output one or more static images and/or one or more animatedvideos. The static image(s) and/or the animated video(s) include one ormore visual representations of the first material 112 and/or the secondmaterial 114. The visual content generation system 600 may transmit thestatic image(s) and/or the animated video(s) to the client computingdevice 140 for display to the artist 142. The artist 142 may use thestatic image(s) and/or the animated video(s) to view the visualrepresentations of the first material 112 and/or the second material 114and may make further adjustments to the parameter values 144. Then, themotion simulation system 120 may output new values of the motionparameters 110, which the visual content generation system 600 may useto output new versions of the static image(s) and/or the animatedvideo(s) that may be viewed by the artist 142 on the client computingdevice 140, or an external computing device (not shown). This processmay be repeated until the artist 142 is satisfied with the appearance ofthe first material 112 and/or the second material 114.

As disclosed above, the client computing device 140 is configured tocommunicate with the motion simulation system 120. For example, theartist 142 may use the client computing device 140 to specify theparameter values 144 to the motion simulation system 120. Optionally,the motion simulation system 120 may be configured to display the firstmaterial 112 and/or the second material 114 to the artist 142 on theclient computing device 140 so that the artist 142 may adjust theparameter values 144 as desired before the values of the motionparameters 110 are input into the visual content generation system 600(see FIG. 6). As described above, the client computing device 140 isconfigured to receive the static image(s) and/or the animated video(s)from the visual content generation system 600 (see FIG. 5) and displaythe static image(s) and/or the animated video(s) to the artist 142 sothat the artist 142 may adjust the parameter values 144. The clientcomputing device 140 may be implemented using the computer system 700illustrated in FIG. 7.

Referring to FIG. 3, one of the first material 112 and/or the secondmaterial 114 is selected (e.g., by the artist 142 and/or the motionsimulation system 120) as a primary material 302, making the other asecondary material 304. In FIG. 3, for ease of illustration, the primarymaterial 302 (e.g., air) is illustrated as forming a bubble (e.g.,particle) inside the secondary material 304 (e.g., water). The secondarymaterial 304 may be considered generally invisible with respect to theprimary material 302. For example, when observing bubbles moving inwater, the water may be generally invisible with respect to the bubblesbut, the water does influence the motion of the bubbles. Similarly, whenobserving a waterfall (not shown), the air may be generally invisiblewith respect to the water but, the air does influence the motion of thewater. Thus, while some processes might treat the secondary material 304as being less visually significant than the primary material 302, thesecondary material 304 is physically significant and affects themovement of the primary material 302.

By way of a non-limiting example, the process 200 (see FIG. 2) may beused to simulate a bubble of the primary material 302 at least partiallysubmerged inside the secondary material 304. The process 200 (see FIG.2) may be less computationally expensive than traditional methodsbecause the process 200 does not model an entire volume of the secondarymaterial 304 as a liquid or gas. In some embodiments, the process 200(see FIG. 2) models a band or a layer portion 308 of the secondarymaterial 304 as a liquid or gas and, in doing so, treats an outerportion 306 of the secondary material 304 as if the dynamics of theouter portion 306 are prescribed. The outer portion 306 includes aregion of the secondary material 304 that is too far away from theprimary material 302 to affect the movement of the primary material 302or to be moved by the primary material 302. Thus, the outer portion 306may be conceptualized and/or modeled as having prescribed dynamics. Insome embodiments, the outer portion 306 can be modeled as if theinteraction between the primary material 302 and the layer portion 308does not have any effect on the outer portion 206. Consequently, theouter portion 306 does not move as a result of interaction with theprimary material 302. In this instance, the outer portion 306 of thesecondary material 304 is not included in the simulation, thus enablingefficient simulation of the overall scene by not computing the outerportion 306 in the simulation. On the other hand, the layer portion 308that surrounds at least a portion of the primary material 302 affectsthe movement of the primary material 302, and thus included in thesimulation because the layer portion 308 is directly affected by themovement of the primary material 302. However, the secondary material304 (e.g., water) may have a density that is much larger (e.g., 1000 or10000 times) than the density of the primary material 302 (e.g., airbubbles). Thus, the secondary material 304 may exert greater force onthe primary material 302 (e.g., pushing the primary material 302 around)than the primary material 302 may exert on the secondary material 304.

The motion simulation system 120 (see FIG. 1) may represent thesecondary material 304 as a sparsely modeled outer volume of fluid and aclosely modeled layer portion 308. Since the representation of the outerportion secondary material 304 is sparse (e.g., modeled with a zero orconstant velocity), the entire volume of the secondary material 304 isnot modeled as a gas, liquid, or other fluid (e.g., with FLIP or AffineParticle in Cell (“APIC”) particles). In other words, instead ofmodeling the entire surrounding volume per se (e.g., a pool) as a fluid,only the layer portion 308 and the primary material 302 need to beconsidered when solving for the movement of the primary material 302 andthe secondary material 304. Thus, the process 200 allows the motionsimulation system 120 (see FIG. 1) to solve a single set of equationsincluding both the layer portion 308 and the primary material 302 at thesame time to obtain the values of the motion parameters 110 (see FIG. 1)more efficiently than prior art methods.

FIG. 2 is a flowchart of the process 200 that may be executed by thesystem 100 of FIG. 1 and used to generate the values of those of themotion parameters 110 that govern the motion of the primary material 302and the layer portion 308 of the secondary material 304 (see FIG. 3).Referring to FIG. 2, in first block 205, the motion simulation system120 (see FIG. 1) represents the primary material 302 (e.g., air) as aplurality of first phase particles (e.g., FLIP or APIC particles). Thefirst phase particles may be implemented as Lagrangian particles. Eachof the first phase particles has an initial position (e.g., with respectto an Eulerian grid). By representing the primary material 302 with thefirst phase particles, the motion simulation system 120 (see FIG. 1) maytrack the first phase particles, which provide satisfactory accuracy fortracking and ensure volume conservation.

In block 210, the motion simulation system 120 identifies a thickness324 of the layer portion 308. Both the layer portion 308 and thethickness 324 are defined between first and second boundaries 320 and322. The first boundary 320 is an interface between the layer portion308 and the primary material 302. The second boundary 322 is an outersurface of the layer portion 308 and may be characterized as being aninterface between the layer portion 308 and the outer portion 306. Insome embodiments, the thickness 324 of the layer portion 308 can beproportional to the size (e.g., diameter) of the bubble or particle ofthe primary material 302. In some embodiments, the thickness 324 can beabout 0.1 to about 10000 times the size (or average size if there are aplurality of bubbles or particles) of the bubble or particle of theprimary material 302. For example, the thickness 324 can be about 0.1,0.2, 0.5, 0.7, 0.8, 1, 2, 5, 10, 15, 20, 25, 50, 100, 200, 500, 1000,2000, 5000, or 10000 times, inclusive of a range between any two sizeslisted therein, of the size (or average size of bubbles or particles) ofthe bubble or particle of the primary material 302. In some embodiments,the thickness 324 can be between about 0.1 and 10000 times, betweenabout 10 and 1000 times, or between about 1 and 100 times, of the size(or average size of bubbles or particles) of the bubble or particle ofthe primary material 302. In various embodiments, a thickness 324 orvolume of the layer portion 308 may depend on the density of thesecondary material 304, the difference in the densities between theprimary material 302 and the secondary material 304, the temperature,humidity, pressure, etc. of the environment, or the like.

In accordance with various embodiments, the thicker that the layerportion 308 is, the closer the simulation results are to beingphysically accurate, with a thinner layer portion leading to dampeningeffects. Therefore, in some embodiments, the thickness 324 of the layerportion 308 represents a trade-off and may be determined by the artist142 (see FIG. 1). In some embodiments, the parameter values 144 (seeFIG. 1) may include the thickness 324.

Then, in block 215 (see FIG. 2), the motion simulation system 120 (seeFIG. 1) represents the layer portion 308 as a second phaserepresentation. The second phase representation may be a sparse Eulerianvolume. Thus, the secondary material 304 may be reduced to a sparseEulerian volume including only the layer portion 308. The second phaserepresentation may include a plurality of voxels (e.g., Eulerian voxelsarranged in an Eulerian grid). One or more attribute (e.g., velocity)may be associated with each voxel. The motion simulation system 120 (seeFIG. 1) may disregard compressibility of the primary material 302 andthe secondary material 304 for efficiency reasons. In other words, themotion simulation system 120 (see FIG. 1) may model both of the primarymaterial 302 and the secondary material 304 as incompressible.

In block 220, the motion simulation system 120 assigns one or moreboundary conditions to the first boundary 320 and/or the second boundary322. For example, when the motion simulation system 120 is simulating abubble of the primary material 302 (e.g., air) positioned inside thesecondary material 304 (e.g., water), the motion simulation system 120may assign a pressure boundary condition to each point along the secondboundary 322. The motion simulation system 120 uses the second boundary322 to enforce the pressure boundary condition(s), which model theprescribed dynamics of the outer portion 306 on the second boundary 322.For example, the motion simulation system 120 may enforce a pressureboundary condition at each point along the second boundary 322. Themotion simulation system 120 may enforce different pressure boundaryconditions at different points along the second boundary 322.Alternatively, the motion simulation system 120 may enforce the samepressure boundary condition at all of the points along the secondboundary 322. The pressure boundary condition(s) is/are assigned to thesecond boundary 322 independently of the type of coupling (e.g.,weak/iterative, strong, and the like) used by the motion simulationsystem 120.

The pressure boundary condition(s) may include hydrostatic pressurevalues. For example, the pressure boundary condition(s) may beimplemented as a hydrostatic pressure field that samples a hydrostaticpressure value for each position in the simulation. The hydrostaticpressure values may be calculated using Equation 1 below, in which avariable “h” represents an evaluation height, a variable “ρ_(w)”represents the density of the secondary material 304, and a variable “g”represents the acceleration of gravity.

$\begin{matrix}{{{p\_ hydrostatic}(h)} = {\rho_{w}{gh}}} & \left( {{Eqn}.\mspace{14mu} 1} \right)\end{matrix}$

When the motion simulation system 120 enforces the pressure boundarycondition(s) (which may be hydrostatic), as opposed to modeling theouter portion 306 as a solid, an apparent sliding effect of the primarymaterial 302 may be reduced. By using the pressure boundary condition(s)(e.g., the hydrostatic pressure values), the motion simulation system120 might also avoid null-modes in a Poisson pressure solve when thePoisson pressure solve is used.

By way of a non-limiting example, the motion simulation system 120 mayuse the hydrostatic pressure values to produce convincing rising bubbleeffects. As the layer portion 308 (e.g., water) moves around the primarymaterial 302 (e.g., the bubble), the hydrostatic pressure values at thesecond boundary 322 push on the primary material 302 and the layerportion 308 (e.g., pushing the primary material 302 and the layerportion 308 upwardly).

The pressure boundary condition(s) may be characterized as representingthe physical effects of the entire outer portion 306 on the primarymaterial 302 and the layer portion 308. In other words, the pressureboundary condition(s) act as an invisible force that affects (e.g.,holds up, shapes, etc.) the primary material 302 and the layer portion308.

Traditional simulations may produce a pressure field for the secondarymaterial 304. When such pressure field includes the primary material 302(e.g., bubbles) embedded in the secondary material 304, the pressurefield may be used to determine the pressure boundary condition(s) intraditional simulations. For example, the hydrostatic pressure valuesalong the second boundary 322 may be calculated from those pressuresoutside the second boundary 322.

After the motion simulation system 120 enforces the pressure boundarycondition(s) on the second boundary 322, the primary material 302 andthe layer portion 308 form a closed system. Because the representationof the secondary material 304 is sparse, the motion simulation system120 (see FIG. 1) may focus computational resources on those componentsthat are important to the simulation, namely, the primary material 302(e.g., bubbles) and the layer portion 308. This may allow the system 100to achieve never-before seen simulation detail.

In block 225, the motion simulation system 120 obtains the values of themotion parameters 110. To obtain the values of the motion parameters 110the motion simulation system 120 strongly or weakly couples the layerportion 308 and the primary material 302 together. When the motionsimulation system 120 (see FIG. 1) solves a single set of equationsincluding both the layer portion 308 and the primary material 302 at thesame time to obtain the values of the motion parameters 110 (see FIG.1), the motion simulation system 120 (see FIG. 1) strongly couples thelayer portion 308 and the primary material 302 together. When the motionsimulation system 120 (see FIG. 1) uses strong coupling, the firstboundary 320 is treated in a standard two-phase Eulerian Navier-Stokesway. For example, the motion simulation system 120 (see FIG. 1) mayenforce pressure jump and velocity continuity conditions. In someimplementations, strong coupling may be computationally expensivebecause of the simultaneous solve for the layer portion 308 and theprimary material 302. Therefore, in some cases, the motion simulationsystem 120 may weakly couple the layer portion 308 and the primarymaterial 302 together, which is less accurate but also lesscomputationally expensive.

When the motion simulation system 120 (see FIG. 1) uses weak coupling(e.g., see Waterfall embodiment described below), the primary andsecondary materials 302 and 304 are modeled separately. In other words,the motion simulation system 120 alternates between solving for thelayer portion 308 and solving for the primary material 302 separately.While alternating, the motion simulation system 120 performs an explicitinteractions exchange (e.g., a drag force exchange 400 illustrated inFIG. 4).

The values of the motion parameters 110 may include at least onevelocity field, which indicates how the primary material 302 and thelayer portion 308 move with respect to their current positions. Forexample, the motion simulation system 120 may obtain a first velocityfield for the primary material 302 (represented by the first phaseparticles) and a second velocity field for the layer portion 308(represented by the second phase representation). Each velocity fieldmay include a vector for each position in the simulation (e.g., theEulerian grid) that indicates how the environment effects the motion(e.g., direction and speed) of a portion of the material currently inthat position.

The motion simulation system 120 may calculate new material states forthe first phase particles as well as new material states for the secondphase representation. The new material states of the first phaseparticles may include positions and attributes of the first phaseparticles. The new positions may be based at least in part on thecurrent positions of the first phase particles and the first velocityfield. At least some of the new positions may be modified (e.g., by themotion simulation system 120), if necessary, using additional processingknown in the art. The new material states of the second phaserepresentation may include states of the Eulerian voxels (in theEulerian grid) and may be based at least in part on the second velocityfield. The new material states of the first phase particles and thesecond phase representation may be included in the values of the motionparameters 110.

The motion simulation system 120 may identify new locations for thefirst and second boundaries 320 and 322 (see FIG. 3) based at least inpart on the new material states (e.g., new positions) for the firstphase particles. Additionally, the motion simulation system 120 maydetermine new pressure boundary condition(s) (e.g., an updatedhydrostatic pressure field) based at least in part on the new locationof the second boundary 322. For example, the motion simulation system120 may calculate the new pressure boundary condition(s) using Equation1 above. The values of the motion parameters 110 may include the newlocations of the first and second boundaries 320 and 322 (see FIG. 3).

Referring to FIG. 3, the type of solve performed by the motionsimulation system 120 in block 225 may depend on the type of couplingneeded to achieve a satisfactory visual result. For example, if thevisual result that is achievable by weak coupling is satisfactory, themotion simulation system 120 may perform separate solves for the primarymaterial 302 and the layer portion 308 and couple these solutionstogether (e.g., as illustrated in FIG. 4). On the other hand, if thevisual result that is achievable with weak coupling is unsatisfactory,the motion simulation system 120 may perform a two-phase pressure solvethat solves for the primary material 302 and the layer portion 308 atthe same time and strongly couples the primary material 302 and thelayer portion 308 together. The two-phase pressure solve may include anincompressible two-phase Navier-Stokes solve on an Eulerian grid (alsoreferred to as a two-phase incompressible ghost-fluid Eulerian solve).Methods of performing two-phase pressure solves are known in the art andneed not be described in detail.

As explained above, the primary material 302 (e.g., an air phase) isrepresented as the first phase particles, which facilitates volumeconservation and accurate tracking of the new locations of the firstboundary 320 (see FIG. 3) and/or the second boundary 322 (see FIG. 3).Thus, unlike the method described in Boyd et al., the motion simulationsystem 120 tracks the first phase particles and recovers the newlocations of the first boundary 320 (see FIG. 3) and/or the secondboundary 322 (see FIG. 3).

In decision block 230, the motion simulation system 120 determineswhether the simulation has completed. The decision in decision block 230is “YES,” when the motion simulation system 120 determines thesimulation has completed. Otherwise, the decision in decision block 230is “NO.” By way of a non-limiting example, blocks 220-235 may berepeated a desired number of iterations (e.g., five times). The numberof iterations might be specified by an artist (e.g., the artist 142) oroperator in advance. For example, blocks 220-235 may be repeated anumber of times required to generate the values of the motion parameters110 needed to create a desired number of frames.

When the decision in decision block 230 is “NO,” the motion simulationsystem 120 advances to block 235 whereat the motion simulation system120 advances the simulation in time. Then, the motion simulation system120 returns to block 220 and assigns the new pressure boundarycondition(s) to the second boundary 322 (see FIG. 3).

When the decision in decision block 230 is “YES,” in block 240, themotion simulation system 120 forwards the values of the motionparameters 110 to an animation creation system, such as the animationcreation system 630 (see FIGS. 1 and 6), which is a component of thevisual content generation system 600 (see FIG. 6), which uses the valuesof the motion parameters 110 to create visual representations of thefirst material 112 and/or the second material 114. Then, the process 200terminates.

By way of a non-limiting example, the process 200 may be used tosimulate a waterfall. In this example, referring to FIG. 3, the primarymaterial 302 is water, water particles, droplets, or mists. Thesecondary material 304 is air that surrounds the water, water particles,droplets, or mists.

As described above, in block 205 (see FIG. 2), the motion simulationsystem 120 (see FIG. 1) represents the primary material 302 (e.g.,water) as the first phase particles (e.g., FLIP or APIC particles) eachhaving an initial position (e.g., with respect to an Eulerian grid).

Then, in block 210 (see FIG. 2), the motion simulation system 120identifies the thickness 324 of the layer portion 308.

Next, in block 215 (see FIG. 2), the motion simulation system 120 (seeFIG. 1) represents the layer portion 308 as the second phaserepresentation. The second phase representation may be a sparse Eulerianvolume. Thus, the motion simulation system 120 (see FIG. 1) mayrepresent the secondary material 304 as a sparse Eulerian volume offluid that includes only the layer portion 308.

In block 220 (see FIG. 2), the motion simulation system 120 (see FIG. 1)assigns boundary condition(s) to the first and second boundaries 320 and322. When solving the first set of equations for the primary material302, the motion simulation system 120 applies a free surface boundarycondition 406 on the first boundary 320 and the drag force 404 in thevicinity of and/or on the second boundary 322. Together, the freesurface boundary condition 406 and the drag force 404 model the layerportion 308 as having one or more prescribed velocities. In other words,the motion simulation system 120 assumes the velocity of the layerportion 308 is prescribed. As described below, the motion simulationsystem 120 may define the drag force 404 based on the velocity of thelayer portion 308 (e.g., stored in a second velocity field) and aerationvalues (described below). When solving the second set of equations forthe layer portion 308, the motion simulation system 120 applies thepressure boundary condition(s) discussed above on the second boundary322 and one or more solid boundary conditions 402 (see FIG. 4) on thefirst boundary 320. In FIG. 4, the pressure boundary condition(s) areillustrated as pressure boundary condition(s) 408. The motion simulationsystem 120 assumes the velocity of the primary material 302 isprescribed (e.g., by using the most recently calculated first velocityfield 504 illustrated in FIG. 5). As described below, the motionsimulation system 120 determines the solid boundary condition(s) 402based on the velocity of the primary material 302 (e.g., stored in thefirst velocity field 504). The solid boundary condition(s) 402 and thepressure boundary condition(s) 408 may be characterized as being “hardconstraints” on velocity and pressure, respectively. The motionsimulation system 120 applies the solid boundary condition(s) 402 andthe pressure boundary condition(s) 408 at the first and secondboundaries 320 and 322, respectively, while solving the second set ofequations.

In block 225 (see FIG. 2), the motion simulation system 120 solves thefirst set of equations for the primary material 302 to obtain the firstvelocity field 504 (see FIG. 5), and the second set of equations for thesecondary material 304 to obtain the second velocity field (not shown),separately. For each position being simulated (e.g., each position ofthe Eulerian grid), the first velocity field 504 (see FIG. 5) stores avelocity value (e.g., a vector) that indicates how the environmentaffects the motion (e.g., direction and speed) of the first phaseparticle, if any, currently in that position. For each position beingsimulated (e.g., each position of the Eulerian grid), the secondvelocity field (not shown) stores a velocity value (e.g., a vector) thatindicates how the environment effects the motion (e.g., direction andspeed) of a portion (e.g., a voxel) of the second phase representation,if any, currently in that position. After the first set of equations issolved, the solution may be supplied to the second set of equations andused to obtain the second velocity field (not shown). When the motionsimulation system 120 next solves the first set of equations to obtain anew first velocity field 510, the motion simulation system 120 performsthe drag force exchange 400, which weakly couples the primary material302 and the layer portion 308 together.

For example, the motion simulation system 120 (see FIG. 1) may begin byfirst solving the first set of equations for the primary material 302 toobtain the first velocity field 504 (see FIG. 5). When the motionsimulation system 120 (see FIG. 1) solves the first set of equations,the motion simulation system 120 applies the free surface boundarycondition 406 at each point along the first boundary 320 and the dragforce 404 in the vicinity of or at each point along the second boundary322, which models the influence of the layer portion 308 (e.g., air) onthe primary material 302 (e.g., water). A free surface is a fluidsurface that is subject to a prescribed pressure condition. For example,the prescribed pressure condition may be zero if the secondary material304 (e.g., air) is significantly lighter than the primary material 302(e.g., water). Methods of calculating the free surface boundarycondition 406 are well-known and will not described herein. The dragforce 404 may be computed based on the aeration values and drag forcevalues, which are determined based at least in part on the relativevelocities of the primary material 302 and the layer portion 308. Thefree surface boundary condition 406 and the drag force 404 model thelayer portion 308 as having one or more prescribed velocities.

Then, the motion simulation system 120 (see FIG. 1) may solve the secondset of equations for the layer portion 308 to obtain the second velocityfield. When the motion simulation system 120 (see FIG. 1) solves thesecond set of equations, the motion simulation system 120 applies thepressure boundary condition(s) 408 assigned to the second boundary 322(based on the prescribed dynamics of the outer portion 306), and thesolid boundary condition(s) 402 assigned to the first boundary 320. Thesolid boundary condition(s) 402 exert(s) force on the layer portion 308(e.g., gas), and not on the primary material 302. This makes sense inview of the fact that water is much heavier than air and easily pushesthe air around. Thus, the motion simulation system 120 applies the solidboundary condition(s) 402 to the layer portion 308 (e.g., gas). Thelayer portion 308 reacts to the solid boundary condition(s) 402 by beingpushed around by the primary material 302 (e.g., fluid). The solidboundary condition(s) 402 is based at least in part on the primarymaterial 302. For example, the solid boundary condition(s) 402 may bebased at least in part on the first velocity field 504 (see FIG. 5).Thus, in this step, because the primary material 302 significantlyinfluences the layer portion 308 (e.g., because water is much heavierthan air), the motion simulation system 120 treats the primary material302 as a solid boundary with a prescribed velocity (e.g., the firstvelocity field 504 illustrated in FIG. 5) that was computed when theprimary material 302 solved the first set of equations. As mentionedabove, the motion simulation system 120 (see FIG. 1) assigns the solidboundary condition(s) 402 (see FIG. 4) to the first boundary 320 (seeFIG. 3).

Thus, for each iteration, the motion simulation system 120 (see FIG. 1)solves for the primary material 302, assuming the velocity of the layerportion 308 is prescribed, and solves for the layer portion 308,assuming the velocity of the primary material 302 is prescribed.

Alternating the solves for the primary material 302 and the layerportion 308 is a weaker coupling scheme than the two-phase solvercoupling scheme discussed above and may be configured to allow theamount of interaction between the air and the water to be at leastpartially artist directed. This weaker coupling scheme may achievebelievable breakup of the water into wispy patterns but may not preservethe shape of bubbles underwater. Thus, depending upon the implementationdetails, the drag force exchange 400 may not be suitable for simulatingbubbles submerged in a fluid (e.g., water).

As mentioned above, the solid boundary condition(s) 402 applies one ormore prescribed velocities to the layer portion 308 (e.g., gas). At thesame time, the layer portion 308 (e.g., gas) exerts the drag force 404on the primary material 302 (e.g., fluid). The drag force 404 may bestored in an adjusted drag force field 502 (see FIG. 5) having a valueat each position being simulated (e.g., each position of the Euleriangrid).

The drag force 404 may be determined based at least in part on thematerial properties of the primary material 302 (e.g., fluid) and a dragforce applied by the layer portion 308 to the primary material 302 (andcalculated based at least in part on the second velocity field).Examples of such material properties include a velocity property, aposition property, and an aeration property. The first velocity field504 (see FIG. 5) stores values of the velocity property. The velocityvalues in the first velocity field 504 may be vectors indicating both adirection and rate of motion. Referring to FIG. 5, the motion simulationsystem 120 may create an aeration field 506. For each position beingsimulated (e.g., each position of the Eulerian grid), the aeration field506 may store an aeration value that indicates how aerated the primarymaterial 302 should be at that position. The aeration values may bedetermined using any method know in the art (e.g., using an aerationheuristic) that measures how aerated the primary material 302 should beat each position being simulated (e.g., each position of the Euleriangrid). Methods of determining the aeration values are known in the artand will not be described herein.

Next, the motion simulation system 120 may create a drag force field 508(see FIG. 5). For each position being simulated (e.g., each position ofthe Eulerian grid), the drag force field 508 (see FIG. 5) stores a dragforce value (e.g., a vector) that indicates how the layer portion 308effects the motion (e.g., direction and speed) of the first phaseparticle, if any, currently in that position. Methods of determining thedrag force values are known in the art and will not be described herein.For example, the drag force value may be computed based at least in parton the relative velocities of the primary material 302 and the layerportion 308.

The drag force 404 (see FIG. 4) may be determined as a function of theaeration field 506 and the drag force field 508. As illustrated in FIG.4 by a dashed line 410, the drag force comes from the secondary material304 and, as illustrated by a dashed line 412, the aeration propertycomes from the primary material 302. The motion simulation system 120may obtain the adjusted drag force field 502 by multiplying each valuein the drag force field 508 by the value in the aeration field 506obtained for the same position.

Then, a new first velocity field 510 may be determined as a function ofthe adjusted drag force field 502 and the previous first velocity field504. For example, the motion simulation system 120 may obtain the newfirst velocity field 510 by multiplying each value in the first velocityfield 504 by the value in the adjusted drag force field 502 obtained forthe same position. The values in the new first velocity field 510 may bevectors indicating both a direction and rate of motion. The values ofthe motion parameters 110 may include the new first velocity field 510,which indicates where the first phase particles representing the primarymaterial 302 move and how quickly.

The motion simulation system 120 may calculate new material states forthe first phase particles as well as new material states for the secondphase representation. The new material states of the first phaseparticles may include positions and attributes of the first phaseparticles. The new positions may be based at least in part on thecurrent positions of the first phase particles and the new firstvelocity field 510. At least some of these new positions may be modified(e.g., by the motion simulation system 120), if necessary, usingadditional processing known in the art. The new material states of thesecond phase representation may include states of Eulerian voxels andmay be based at least in part on the second velocity field (not shown).The new material states of the first phase particles and the secondphase representation may be included in the values of the motionparameters 110.

The motion simulation system 120 may identify new locations for thefirst and second boundaries 320 and 322 (see FIG. 3) based at least inpart on the new material states (e.g., new positions) of the first phaseparticles. Additionally, the motion simulation system 120 may determinenew pressure boundary condition(s) (e.g., an updated hydrostaticpressure field) based at least in part on the new location of the secondboundary 322. For example, the motion simulation system 120 maycalculate the new pressure boundary condition(s) using Equation 1 above.The values of the motion parameters 110 may include the new locations ofthe first and second boundaries 320 and 322 (see FIG. 3).

As explained above, referring to FIG. 3, the more aerated the primarymaterial 302 is, the greater the drag force 404 (see FIG. 4) is. Thus,the aeration property may be used to modulate how much drag force isapplied to the primary material 302 by the layer portion 308 of thesecondary material 304.

Optionally, the values of the first velocity field 504, the aerationfield 506, the drag force field 508, and/or the new first velocity field510 may be modified (e.g., multiplied) by one or more additional values.Examples of such the additional value(s) include density of the primarymaterial 302, artistic or artist-controlled parameters, and/or the like.

In decision block 230, the motion simulation system 120 determineswhether the simulation has completed. The decision in decision block 230is “YES,” when the motion simulation system 120 determines thesimulation has completed. Otherwise, the decision in decision block 230is “NO.” By way of a non-limiting example, blocks 220-235 may berepeated a desired number of iterations (e.g., five times). The numberof iterations might be specified by an artist (e.g., the artist 142) oroperator in advance. For example, blocks 220-235 may be repeated anumber of times required to generate the values of the motion parameters110 needed to create a desired number of frames.

When the decision in decision block 230 is “NO,” the motion simulationsystem 120 advances to block 235 whereat the motion simulation system120 advances the simulation in time. Then, the motion simulation system120 returns to block 220 and assigns new boundary condition(s) to thefirst and second boundaries 320 and 322.

When the decision in decision block 230 is “YES,” in block 240, themotion simulation system 120 forwards the values of the motionparameters 110 to the animation creation system 630 (see FIGS. 1 and 6)component of the visual content generation system 600 (see FIG. 6),which uses the values of the motion parameters 110 to create visualrepresentations of a waterfall including the first material 112 and/orthe second material 114. Then, the process 200 terminates.

For example, FIG. 6 illustrates the example visual content generationsystem 600 as might be used to generate imagery in the form of stillimages and/or video sequences of images. Visual content generationsystem 600 might generate imagery of live action scenes, computergenerated scenes, or a combination thereof. In a practical system, usersare provided with tools that allow them to specify, at high levels andlow levels where necessary, what is to go into that imagery. Forexample, a user might be an animation artist (like artist 142illustrated in FIG. 1) and might use visual content generation system600 to capture interaction between two human actors performing live on asound stage and replace one of the human actors with acomputer-generated anthropomorphic non-human being that behaves in waysthat mimic the replaced human actor's movements and mannerisms, and thenadd in a third computer-generated character and background sceneelements that are computer-generated, all in order to tell a desiredstory or generate desired imagery.

Still images that are output by visual content generation system 600might be represented in computer memory as pixel arrays, such as atwo-dimensional array of pixel color values, each associated with apixel having a position in a two-dimensional image array. Pixel colorvalues might be represented by three or more (or fewer) color values perpixel, such as a red value, a green value, and a blue value (e.g., inRGB format). Dimensions of such a two-dimensional array of pixel colorvalues might correspond to a preferred and/or standard display scheme,such as 1920-pixel columns by 1280-pixel rows or 4096-pixel columns by2160-pixel rows, or some other resolution. Images might or might not bestored in a compressed format, but either way, a desired image may berepresented as a two-dimensional array of pixel color values. In anothervariation, images are represented by a pair of stereo images forthree-dimensional presentations and in other variations, an imageoutput, or a portion thereof, might represent three-dimensional imageryinstead of just two-dimensional views. In yet other embodiments, pixelvalues are data structures and a pixel value is associated with a pixeland can be a scalar value, a vector, or another data structureassociated with a corresponding pixel. That pixel value might includecolor values, or not, and might include depth values, alpha values,weight values, object identifiers or other pixel value components.

A stored video sequence might include a plurality of images such as thestill images described above, but where each image of the plurality ofimages has a place in a timing sequence and the stored video sequence isarranged so that when each image is displayed in order, at a timeindicated by the timing sequence, the display presents what appears tobe moving and/or changing imagery. In one representation, each image ofthe plurality of images is a video frame having a specified frame numberthat corresponds to an amount of time that would elapse from when avideo sequence begins playing until that specified frame is displayed. Aframe rate might be used to describe how many frames of the stored videosequence are displayed per unit time. Example video sequences mightinclude 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other framerates. In some embodiments, frames are interlaced or otherwise presentedfor display, but for clarity of description, in some examples, it isassumed that a video frame has one specified display time, but othervariations might be contemplated.

One method of creating a video sequence is to simply use a video camerato record a live action scene, i.e., events that physically occur andcan be recorded by a video camera.

The events being recorded can be events to be interpreted as viewed(such as seeing two human actors talk to each other) and/or can includeevents to be interpreted differently due to clever camera operations(such as moving actors about a stage to make one appear larger than theother despite the actors actually being of similar build, or usingminiature objects with other miniature objects so as to be interpretedas a scene containing life-sized objects).

Creating video sequences for story-telling or other purposes often callsfor scenes that cannot be created with live actors, such as a talkingtree, an anthropomorphic object, space battles, and the like. Such videosequences might be generated computationally rather than capturing lightfrom live scenes. In some instances, an entirety of a video sequencemight be generated computationally, as in the case of acomputer-animated feature film. In some video sequences, it is desirableto have some computer-generated imagery and some live action, perhapswith some careful merging of the two.

While computer-generated imagery might be creatable by manuallyspecifying each color value for each pixel in each frame, this is likelytoo tedious to be practical. As a result, a creator uses various toolsto specify the imagery at a higher level. As an example, an artist(e.g., artist 142 illustrated in FIG. 1) might specify the positions ina scene space, such as a three-dimensional coordinate system, of objectsand/or lighting, as well as a camera viewpoint, and a camera view plane.From that, a rendering engine could take all of those as inputs, andcompute each of the pixel color values in each of the frames. In anotherexample, an artist specifies position and movement of an articulatedobject having some specified texture rather than specifying the color ofeach pixel representing that articulated object in each frame.

In a specific example, a rendering engine performs ray tracing wherein apixel color value is determined by computing which objects lie along aray traced in the scene space from the camera viewpoint through a pointor portion of the camera view plane that corresponds to that pixel. Forexample, a camera view plane might be represented as a rectangle havinga position in the scene space that is divided into a grid correspondingto the pixels of the ultimate image to be generated, and if a raydefined by the camera viewpoint in the scene space and a given pixel inthat grid first intersects a solid, opaque, blue object, that givenpixel is assigned the color blue. Of course, for moderncomputer-generated imagery, determining pixel colors—and therebygenerating imagery—can be more complicated, as there are lightingissues, reflections, interpolations, and other considerations.

As illustrated in FIG. 6, a live action capture system 602 captures alive scene that plays out on a stage 604. Live action capture system 602is described herein in greater detail, but might include computerprocessing capabilities, image processing capabilities, one or moreprocessors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown.

In a specific live action capture system, cameras 606(1) and 606(2)capture the scene, while in some systems, there might be other sensor(s)608 that capture information from the live scene (e.g., infraredcameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).On stage 604, there might be human actors, animal actors, inanimateobjects, background objects, and possibly an object such as a greenscreen 610 that is designed to be captured in a live scene recording insuch a way that it is easily overlaid with computer-generated imagery.Stage 604 might also contain objects that serve as fiducials, such asfiducials 612(1)-(3), that might be used post-capture to determine wherean object was during capture. A live action scene might be illuminatedby one or more lights, such as an overhead light 614.

During or following the capture of a live action scene, live actioncapture system 602 might output live action footage to a live actionfootage storage 620. A live action processing system 622 might processlive action footage to generate data about that live action footage andstore that data into a live action metadata storage 624. Live actionprocessing system 622 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Live action processing system 622 might processlive action footage to determine boundaries of objects in a frame ormultiple frames, determine locations of objects in a live action scene,where a camera was relative to some action, distances between movingobjects and fiducials, etc. Where elements have sensors attached to themor are detected, the metadata might include location, color, andintensity of overhead light 614, as that might be useful inpost-processing to match computer-generated lighting on objects that arecomputer-generated and overlaid on the live action footage. Live actionprocessing system 622 might operate autonomously, perhaps based onpredetermined program instructions, to generate and output the liveaction metadata upon receiving and inputting the live action footage.The live action footage can be camera-captured data as well as data fromother sensors.

An animation creation system 630 is another part of visual contentgeneration system 600. Animation creation system 630 might includecomputer processing capabilities, image processing capabilities, one ormore processors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown. Animation creationsystem 630 might be used by animation artists, managers, and others tospecify details, perhaps programmatically and/or interactively, ofimagery to be generated. From user input and data from a database orother data source, indicated as a data store 632, animation creationsystem 630 might generate and output data representing objects (e.g., ahorse, a human, a ball, a teapot, a cloud, a light source, a texture,etc.) to an object storage 634, generate and output data representing ascene into a scene description storage 636, and/or generate and outputdata representing animation sequences to an animation sequence storage638.

Scene data might indicate locations of objects and other visualelements, values of their parameters, lighting, camera location, cameraview plane, and other details that a rendering engine 650 might use torender CGI imagery. For example, scene data might include the locationsof several articulated characters, background objects, lighting, etc.specified in a two-dimensional space, three-dimensional space, or otherdimensional space (such as a 2.5-dimensional space, three-quarterdimensions, pseudo-3D spaces, etc.) along with locations of a cameraviewpoint and view place from which to render imagery. For example,scene data might indicate that there is to be a red, fuzzy, talking dogin the right half of a video and a stationary tree in the left half ofthe video, all illuminated by a bright point light source that is aboveand behind the camera viewpoint. In some cases, the camera viewpoint isnot explicit, but can be determined from a viewing frustum. In the caseof imagery that is to be rendered to a rectangular view, the frustumwould be a truncated pyramid. Other shapes for a rendered view arepossible and the camera view plane could be different for differentshapes.

Animation creation system 630 might be interactive, allowing a user toread in animation sequences, scene descriptions, object details, etc.and edit those, possibly returning them to storage to update or replaceexisting data. As an example, an operator might read in objects fromobject storage into a baking processor 642 that would transform thoseobjects into simpler forms and return those to object storage 634 as newor different objects. For example, an operator might read in an objectthat has dozens of specified parameters (movable joints, color options,textures, etc.), select some values for those parameters and then save abaked object that is a simplified object with now fixed values for thoseparameters.

Rather than requiring user specification of each detail of a scene, datafrom data store 632 might be used to drive object presentation. Forexample, if an artist is creating an animation of a spaceship passingover the surface of the Earth, instead of manually drawing or specifyinga coastline, the artist might specify that animation creation system 630is to read data from data store 632 in a file containing coordinates ofEarth coastlines and generate background elements of a scene using thatcoastline data.

Animation sequence data might be in the form of time series of data forcontrol points of an object that has attributes that are controllable.For example, an object might be a humanoid character with limbs andjoints that are movable in manners similar to typical human movements.An artist can specify an animation sequence at a high level, such as“the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) overtime T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5degrees per frame”) or even at a very high level (e.g., “character Ashould move, consistent with the laws of physics that are given for thisscene, from point P1 to point P2 along a specified path”).

Animation sequences in an animated scene might be specified by whathappens in a live action scene. An animation driver generator 644 mightread in live action metadata, such as data representing movements andpositions of body parts of a live actor during a live action scene.Animation driver generator 644 might generate corresponding animationparameters to be stored in animation sequence storage 638 for use inanimating a CGI object. This can be useful where a live action scene ofa human actor is captured while wearing mo-cap fiducials (e.g.,high-contrast markers outside actor clothing, high-visibility paint onactor skin, face, etc.) and the movement of those fiducials isdetermined by live action processing system 622. Animation drivergenerator 644 might convert that movement data into specifications ofhow joints of an articulated CGI character are to move over time.

A rendering engine 650 can read in animation sequences, scenedescriptions, and object details, as well as rendering engine controlinputs, such as a resolution selection and a set of renderingparameters. Resolution selection might be useful for an operator tocontrol a trade-off between speed of rendering and clarity of detail, asspeed might be more important than clarity for a movie maker to testsome interaction or direction, while clarity might be more importantthan speed for a movie maker to generate data that will be used forfinal prints of feature films to be distributed. Rendering engine 650might include computer processing capabilities, image processingcapabilities, one or more processors, program code storage for storingprogram instructions executable by the one or more processors, as wellas user input devices and user output devices, not all of which areshown.

Visual content generation system 600 can also include a merging system660 that merges live footage with animated content. The live footagemight be obtained and input by reading from live action footage storage620 to obtain live action footage, by reading from live action metadatastorage 624 to obtain details such as presumed segmentation in capturedimages segmenting objects in a live action scene from their background(perhaps aided by the fact that green screen 610 was part of the liveaction scene), and by obtaining CGI imagery from rendering engine 650.

A merging system 660 might also read data from rulesets formerging/combining storage 662. A very simple example of a rule in aruleset might be “obtain a full image including a two-dimensional pixelarray from live footage, obtain a full image including a two-dimensionalpixel array from rendering engine 650, and output an image where eachpixel is a corresponding pixel from rendering engine 650 when thecorresponding pixel in the live footage is a specific color of green,otherwise output a pixel value from the corresponding pixel in the livefootage.”

Merging system 660 might include computer processing capabilities, imageprocessing capabilities, one or more processors, program code storagefor storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Merging system 660 might operate autonomously,following programming instructions, or might have a user interface orprogrammatic interface over which an operator can control a mergingprocess. In some embodiments, an operator can specify parameter valuesto use in a merging process and/or might specify specific tweaks to bemade to an output of merging system 660, such as modifying boundaries ofsegmented objects, inserting blurs to smooth out imperfections, oradding other effects. Based on its inputs, merging system 660 can outputan image to be stored in a static image storage 670 and/or a sequence ofimages in the form of video to be stored in an animated/combined videostorage 672.

Thus, as described, visual content generation system 600 can be used togenerate video that combines live action with computer-generatedanimation using various components and tools, some of which aredescribed in more detail herein. While visual content generation system600 might be useful for such combinations, with suitable settings, itcan be used for outputting entirely live action footage or entirely CGIsequences. The code may also be provided and/or carried by a transitorycomputer readable medium, e.g., a transmission medium such as in theform of a signal transmitted over a network.

According to one embodiment, the techniques described herein areimplemented by one or more generalized computing systems programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Special-purpose computingdevices may be used, such as desktop computer systems, portable computersystems, handheld devices, networking devices or any other device thatincorporates hard-wired and/or program logic to implement thetechniques.

For example, FIG. 7 is a block diagram that illustrates a computersystem 700 upon which the computer systems of the systems describedherein and/or visual content generation system 600 (see FIG. 6) may beimplemented. Computer system 700 includes a bus 702 or othercommunication mechanism for communicating information, and a processor704 coupled with bus 702 for processing information. Processor 704 maybe, for example, a general-purpose microprocessor.

Computer system 700 also includes a main memory 706, such as arandom-access memory (RAM) or other dynamic storage device, coupled tobus 702 for storing information and instructions to be executed byprocessor 704. Main memory 706 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 704. Such instructions, whenstored in non-transitory storage media accessible to processor 704,render computer system 700 into a special-purpose machine that iscustomized to perform the operations specified in the instructions.

Computer system 700 further includes a read only memory (ROM) 708 orother static storage device coupled to bus 702 for storing staticinformation and instructions for processor 704. A storage device 710,such as a magnetic disk or optical disk, is provided and coupled to bus702 for storing information and instructions.

Computer system 700 may be coupled via bus 702 to a display 712, such asa computer monitor, for displaying information to a computer user. Aninput device 714, including alphanumeric and other keys, is coupled tobus 702 for communicating information and command selections toprocessor 704. Another type of user input device is a cursor control716, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor704 and for controlling cursor movement on display 712. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 700 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 700 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 700 in response to processor 704 executing one or more sequencesof one or more instructions contained in main memory 706. Suchinstructions may be read into main memory 706 from another storagemedium, such as storage device 710. Execution of the sequences ofinstructions contained in main memory 706 causes processor 704 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may includenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 710.Volatile media includes dynamic memory, such as main memory 706. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, an EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire, and fiber optics, including thewires that include bus 702. Transmission media can also take the form ofacoustic or light waves, such as those generated during radio-wave andinfra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 704 for execution. For example,the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over anetwork connection. A modem or network interface local to computersystem 700 can receive the data. Bus 702 carries the data to main memory706, from which processor 704 retrieves and executes the instructions.The instructions received by main memory 706 may optionally be stored onstorage device 710 either before or after execution by processor 704.

Computer system 700 also includes a communication interface 718 coupledto bus 702. Communication interface 718 provides a two-way datacommunication coupling to a network link 720 that is connected to alocal network 722. For example, communication interface 718 may be anetwork card, a modem, a cable modem, or a satellite modem to provide adata communication connection to a corresponding type of telephone lineor communications line. Wireless links may also be implemented. In anysuch implementation, communication interface 718 sends and receiveselectrical, electromagnetic, or optical signals that carry digital datastreams representing various types of information.

Network link 720 typically provides data communication through one ormore networks to other data devices. For example, network link 720 mayprovide a connection through local network 722 to a host computer 724 orto data equipment operated by an Internet Service Provider (ISP) 726.ISP 726 in turn provides data communication services through theworld-wide packet data communication network now commonly referred to asthe “Internet” 728. Local network 722 and Internet 728 both useelectrical, electromagnetic, or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 720 and through communication interface 718, which carrythe digital data to and from computer system 700, are example forms oftransmission media.

Computer system 700 can send messages and receive data, includingprogram code, through the network(s), network link 720, andcommunication interface 718. In the Internet example, a server 730 mighttransmit a requested code for an application program through theInternet 728, ISP 726, local network 722, and communication interface718. The received code may be executed by processor 704 as it isreceived, and/or stored in storage device 710, or other non-volatilestorage for later execution.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory. The code may also be provided carried by atransitory computer readable medium e.g., a transmission medium such asin the form of a signal transmitted over a network.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

The use of examples, or exemplary language (e.g., “such as”) providedherein, is intended merely to better illuminate embodiments of theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element as essential to thepractice of the invention.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and combinations, additions, re-arrangements, and the likeare contemplated in alternative embodiments of the present invention.Thus, while the invention has been described with respect to exemplaryembodiments, one skilled in the art will recognize that numerousmodifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

What is claimed is:
 1. A computer-implemented method for simulatinginteractions between two different materials, the computer-implementedmethod comprising: under the control of one or more computer systemsconfigured with executable instructions: modeling a primary material asa plurality of objects; modeling a secondary material as a grid-basedfluid volume completely surrounding the plurality of objects, the fluidvolume comprising a layer portion and an outer portion, the layerportion being positioned between the plurality of objects and the outerportion; assigning at least one boundary condition to a boundarypositioned between the layer portion and the outer portion based atleast in part on pressures of the outer portion and configured to affectthe layer portion, wherein the at least one boundary condition comprisesa pressure boundary condition; determining values of motion parametersfor the primary material based on the plurality of objects and the layerportion; and determining values of motion parameters for the secondarymaterial based on the plurality of objects, the layer portion, the atleast one boundary condition, independent of the outer portion.
 2. Thecomputer-implemented method of claim 1, wherein the layer portion has apredefined thickness.
 3. The computer-implemented method of claim 1,wherein the layer portion has a thickness that depends on a differencein densities between the primary material and the secondary material. 4.The computer-implemented method of claim 1, wherein motion of theobjects of the plurality of objects with respect to one another isconstrained in zero, one, two, or three degrees of freedom.
 5. Thecomputer-implemented method of claim 1, wherein a sparse modeling of theouter portion of the secondary material is used to compute the one ormore boundary conditions.
 6. The computer-implemented method of claim 1,wherein the at least one boundary condition includes a velocity at theboundary configured to affect the layer portion.
 7. Thecomputer-implemented method of claim 6, wherein the at least oneboundary condition includes a two-phase Eulerian Navier-Stokes approachthat allows a pressure jump between the primary material and secondarymaterial, and wherein the at least one boundary condition includesvelocity continuity conditions at the boundary.
 8. Thecomputer-implemented method of claim 1, wherein the at least oneboundary condition allows for coupling between the layer portion and theprimary material.
 9. The computer-implemented method of claim 1, furthercomprising: calculating an aeration field for the primary material; andcalculating a drag force field for the secondary material, wherein theprimary velocity field, the aeration field, and the drag force field areused for calculating a new primary velocity field for the primarymaterial.
 10. The computer-implemented method of claim 1, wherein theboundary is a second boundary, and the at least one boundary conditionis at least one second boundary condition, the method furthercomprising: assigning at least one first boundary condition to a firstboundary positioned between the primary material and the layer portion,the at least one first boundary condition comprising a free surfaceboundary condition.
 11. The computer-implemented method of claim 10,wherein the at least one first boundary condition comprises a drag forcein a vicinity of the first boundary.
 12. The computer-implemented methodof claim 10, wherein the at least one first boundary condition comprisesa solid boundary condition applied at the second boundary and the atleast one primary boundary condition comprises a pressure boundarycondition applied at the first boundary.
 13. The computer-implementedmethod of claim 1, further comprising: calculating a primary velocityfield for the primary material, wherein the primary velocity field isconfigured to store a primary velocity value that indicates how anenvironment affects motion of the plurality of objects.
 14. Thecomputer-implemented method of claim 13, further comprising: calculatinga secondary velocity field for the secondary material, wherein thesecondary velocity field is configured to store a secondary velocityvalue that indicates how the environment affects motion of a portion ofthe secondary material.
 15. The computer-implemented method of claim 1,wherein the primary material comprises air and the plurality of objectscomprises a plurality of bubbles, and wherein the secondary materialcomprises water.
 16. The computer-implemented method of claim 1, whereinthe primary material comprises one of a polar fluid or a non-polarfluid, and the secondary material comprises the other of a polar fluidor a non-polar fluid.
 17. A non-transitory computer-readable storagemedium storing instructions, which when executed by at least oneprocessor of a computer system, cause the computer system to carry outthe method of claim
 1. 18. A computer-readable medium carryinginstructions, which when executed by at least one processor of acomputer system, cause the computer system to carry out the method ofclaim
 1. 19. The computer-implemented method of claim 1, furthercomprising generating one or more visual representations of the primarymaterial interacting with the secondary material based on the values ofthe motion parameters.
 20. The computer-implemented method of claim 17,wherein either the primary material or the secondary material isinvisible in the one or more visual representations.